Quality of Diagnostic Studies

I n this issue of the BJO (p 261), Siddiqui et al review the compliance of researchers with quality standards for evaluations of diagnostic test performance (DTP). ‘‘Standards’’ were originally set by the McMaster evidence based medicine (EBM) group 2 and they have continued to evolve over recent years. Unfortunately, the standards appear to have had little impact since reviews of recent evaluations have shown that they tend to be of poor quality, in medicine generally and in ophthalmology and other specialties. The review of Siddiqui et al confirms this gloomy picture. In contrast, during the same period, there has been substantial improvement in the quality and reporting of evaluations of the effectiveness of treatments. Why has research to evaluate DTP not benefited in a similar way from the EBM ‘‘movement’’? Perhaps improving the quality of research about effectiveness was seen as a priority because it was perceived to be important to patients—the ‘‘bit’’ of health care that makes them better—or because the resources wasted from using treatments that don’t work (and not using ones that do) was much easier for the public and media to appreciate. Perhaps the principles of high quality research to evaluate DTP are more difficult to grasp than controlled experiments to assess effectiveness. Or perhaps we can just blame Archie Cochrane! Whatever the reason, prioritising research about effectiveness might be seen as paradoxical since it is difficult to optimise treatment without first knowing the diagnosis. It is also not clearly justified on an efficiency basis, since substantial (and increasing) amounts of healthcare resources are spent on diagnosis, with new and expensive diagnostic technologies emerging. And the diversity of evidence about DTP is often not appreciated—for example, patients’ responses to standard questions when taking a history and standardised observations of clinical signs all constitute diagnostic ‘‘test’’ information, the value of which can be quantified. The relative neglect of evidence about DTP may, at last, be about to change. The Cochrane Collaboration has long appreciated the importance of such evidence—a methods groups on the topic was registered in 1995—and, in 2003, the collaboration took the decision to develop a new database of systematic reviews of diagnostic test accuracy. This will be developed in parallel with the existing database of systematic reviews of the effectiveness of healthcare interventions. This new review of ophthalmic tests might appear to suggest that things are improving compared to the situation during the 1990s. All evaluations scored some points, with scores ranging from 8–19/25 compared with 0–5/7. However, although all STARD items are important, they are not all equally important. Failure to report some item may mislead a reader but does not necessarily invalidate the evidence. In contrast, poor compliance in reporting particular items leads (on average) to biased, optimistic estimates of DTP. Unfortunately, compliance with these items, about masking/blinding (item 11) and workup bias (item 16), was poorer than for others, with 6/16 and 4/15 papers respectively judged to be compliant with the standard. Reporting indeterminate results (and analysing them correctly) is also crucial, since decisions still need to be made about patients who give such results. Failure to comply will almost always cause researchers to overestimate DTP. This item was poorly reported as well (item 22: 5/16).